Supervised Visual Attention for Simultaneous Multimodal Machine Translation

Recently, there has been a surge in research multimodal machine translation (MMT), where additional modalities such as images are used to improve quality of textual systems. A particular use for systems is the task simultaneous translation, visual context shown complement partial information provided by source sentence, especially early phases translation. In this paper, we propose first Transformer-based MMT architecture, which not previously explored field. Additionally, extend model with an auxiliary supervision signal that guides its attention mechanism using labelled phrase-region alignments. We perform comprehensive experiments on three language directions and conduct thorough quantitative qualitative analyses both automatic metrics manual inspection. Our results show (i) supervised consistently improves models, (ii) fine-tuning loss enabled leads better performance than training from scratch. Compared state-of-the-art, our proposed achieves improvements up 2.3 BLEU 3.5 METEOR points.